This invention relates in general to image systems, and in particular to methods and apparatus for evaluating image systems.
An image system is designed to capture an image of an object and to display the captured image to a user. Image systems are used for a variety of purposes. Many image systems are based on photographic technology including well known silver halide image capture and development techniques of conventional photography. The imaging radiation may be visible light, infrared light, of x-rays. Other image systems are digital in nature and rely upon semiconductor arrays for capturing an image. Digital imaging systems include digital cameras, digital radiography (including but not limited to digital mammography), optical character/image recognition systems, and scanners. Still other imaging systems rely upon dry photography, such as electrophotography.
Any image system must reliably display the captured image. However, it is difficult to measure the reliability and accuracy of an imaging system. It is also difficult to compare on type of image system to another. For example, will a digital mammography system identify tumors as well as, better or worse than a traditional silver halide system. One measuring technique relies upon observer testing where the observer is asked to identify objects in an image. Such techniques are flawed because they fail to account for false positives.
Since 1966 others have used Receiver Operator Characteristic (ROC) for threshold level detection experiments used for making likelihood ratio discriminations between true signal and false signal presentations in detection experiments with noise present. Methods are known for estimating the ROC from detection data. Other researchers have provided a mathematical framework in which it is possible to conceive of detection imaging as a two stage process of visual abstraction followed by a signal evaluation stage). In the early ""70s the ROC method was found to be deficient in radiographic detection tasks because of the identifiability problem, namely:
1) One can never be certain whether a particular response was due to a signal or whether it was spontaneously generated.
2) Even if one were assured that a response was due to some signal, and if the signals come at all close together, one cannot be certain to which signal it was due.
Still others attempted to resolve the identifiability problem by constraining the sampling window and by requiring that the observer specify the location of the detected image. Multiple signal presentations were also used. Such localization experiments were similarly considered as unsuccessful since they had the effect of reducing the scale of the problem but the same identifiability problem was operative at the reduced scale. The multiple signal detection experiments were an improvement but they pointed to the use of a still greater number of signals the number of which should be chosen randomly.
The Free Response Experiment was a great improvement in the experimental design providing for a realistic statistical interpretive framework for radiographic detection experiments. The false positive were assumed to be locally Poisson distributed which makes sense for an imaging process which generates false positive images on randomly generated fields of noise or on structured medical noise. Researchers working with the Radiological Medicine Group at the Mayo Clinic generated thousands of FROC detection data on 5 mm circles embedded in random fields of radiographic mottle. It was estimated the parameters of a FROC curve which was a non-linear relationship between the false positive rate and the corresponding fraction of true positive detected and scored with the appropriate ROC rating.
One researcher and coinventor of this invention, Winton Brown, independently noted that there were potentially serious ill-conditioned inference problems in the estimation procedures for the FROC parameters. This made it impossible to estimate the ROC parameters separate from the imaging rate parameters. Brown concluded that there were two columns of data missing which were necessary to estimate the ROC parameters from the FROC experiment. which were related to visual system affects arising from the deconvolution of image data from the radiographic mottle noise field. (True Negative imaging events and errors in the imaging of the True Positive events). Brown assumed that these were due to statistical behavior arising in the visual search and image abstraction process.. Estimating of the ROC from FROC data would require precision eyetracking, a technology which Dr. Harold Kundell was working with in tracking scan paths and saccades in teaching Radiographic Medicine.
Brown has described the problems with measuring the ROC from Free Response data and provided a formalized theoretical framework for understanding the problem of visual detection in fields of noise. He recommended using precision eyetracking to gather data in Free Response detection experiments.
We have demonstrated the Free Response detection on his Applied Science Lab ASL 4000 SU, video based infrared LED Eyetracker. Our experiment indicated that the full range of imaging events and ROC statistics could readily be generated with the eyetracking instrument. We conducted a Threshold level Detection Experiment that is a comprehensive technique for measuring the image quality of an imaging system. That technique can be used throughout the field of Experimental Radiology and other imaging systems for determining the improvements in system detection performance.
The proposed uses for this technique are in general application of ROC type radiographic detection experiments specifically determining the detection performance as a function of the physical properties of the radiographic system. The technique has several applications:
1) Determine the suitability of the FDA standards for testing to determine the acceptability of a Mammography Radiographic System. (W. Moore, E.K. Co.)
2) Determine the effect of film contrast on the ability of a Radiologist to detect lesions in Mammography. (W. Wolski, E.K. Co.)
3)Determine the capability of computer aided Mammography when compared with human observers.